By IceMimosa

更新日期:

文章目录
  1. Oozie-6_Web Service API
  2. 1. 简介
  3. 2. versions
  4. 3. admin
    1. 3.1 status
    2. 3.2 available-oozie-servers
    3. 3.3 OS Environment
    4. 3.4 Java System Properties
    5. 3.5 Oozie Configuration
    6. 3.6 Oozie Instrumentation 和 Oozie Metrics
    7. 3.7 Build Version
    8. 3.8 Available Time Zones
    9. 3.9 List available sharelib
  5. 4. Job 和 Jobs
    1. 4.1 Standard Job Submission
    2. 4.2 Proxy MapReduce Job Submission
    3. 4.3 Proxy Pig Job Submission
    4. 4.4 Proxy Hive Job Submission
    5. 4.5 Proxy Sqoop Job Submission
    6. 4.6 Managing a Job
      1. 4.6.1 Start a Job
      2. 4.6.2 Re-Runing a Workflow Job
      3. 4.6.3 Re-Runing a coordinator job
    7. 返回：
    8. 返回：
    9. 4.9 Job Log
    10. 4.10 Job graph
    11. 4.11 Jobs Information
    12. 4.12 Jobs information using Bulk API

Oozie-6_Web Service API

[TOC]

1. 简介

The Oozie Web Services API is a HTTP REST JSON API.
All responses are in UTF-8 .

Assuming Oozie is runing at OOZIE_URL ( 如http://localhost:11000/oozie ) , the following web services end points are supported:

  • /versions
  • /v1/admin
  • /v1/job
  • /v1/jobs
  • /v2/job
  • /v2/jobs
  • /v2/admin
  • /v2/sla

在一些案例中，有些相当于使用 oozie 命令行：
https://oozie.apache.org/docs/4.1.0/DG_CommandLineTool.html

2. versions

  • 描述： 获取Oozie服务支持的协议版本号
  • method：GET
  • url：/versions
  • 入参：无
  • 出参：目前返回 0, 1, 2

  • 示例：

    • url：GET /versions

    • 返回：

      1
      [0,1,2]

3. admin

获取系统的 状态 以及一些 配置信息

1
2
# admin支持的 sub-resources
status, os-env, sys-props, configuration, instrumentation, systems, available-timezones .

3.1 status

  • 描述：获取系统服务状态
  • method：GET
  • url：/v1/admin/status
  • 入参：systemmode（可选, 为下面模式的一种)。如果指定, 相应无返回值。
  • 出参：返回系统的状态, 可能值 NORMAL , NOWEBSERVICE , SAFEMODE .

  • 示例：

    • url：GET /v1/admin/status

    • 返回：

      1
      {"systemMode":"NORMAL"}

3.2 available-oozie-servers

  • 描述：获取可用的Oozie服务
  • method：GET
  • url：/v2/admin/available-oozie-servers
  • 入参：无
  • 出参：可用的Oozie服务列表

  • 示例：

    • url：GET /v2/admin/available-oozie-servers

    • 返回：

      1
      2
      3
      4
      5
      {
          "hostA": "http://hostA:11000/oozie",
          "hostB": "http://hostB:11000/oozie",
          "hostC": "http://hostC:11000/oozie",
      }

3.3 OS Environment

  • 描述：获取Oozie服务系统的配置参数
  • method：GET
  • url：/v1/admin/os-env
  • 出参：Content-Type: application/json;charset=UTF-8

3.4 Java System Properties

  • 描述：获取Oozie Java系统参数
  • method：GET
  • url：/v1/admin/java-sys-properties
  • Content-Type: application/json;charset=UTF-8

3.5 Oozie Configuration

  • 描述：获取Oozie服务的配置
  • method：GET
  • url：/v1/admin/configuration
  • 出参：Content-Type: application/json;charset=UTF-8

3.6 Oozie Instrumentation 和 Oozie Metrics

  • 描述：获取Oozie的一些测试指标数据
  • method：GET
  • url：/v1/admin/instrumentation 和 /v2/admin/metrics
  • 出参：Content-Type: application/json;charset=UTF-8

3.7 Build Version

  • 描述：获取Oozie build version
  • method：GET
  • url：/v1/admin/build-version

  • 示例返回：

    1
    {"buildVersion": "4.1.0-cdh5.7.0"}

3.8 Available Time Zones

  • 描述：获取可用的时区
  • method：GET
  • url：/v1/admin/available-timezones
  • 出参：Content-Type: application/json;charset=UTF-8

3.9 List available sharelib

  • 描述：获取共享的libs
  • method：GET
  • url：/v2/admin/list_sharelib
  • 入参：lib（可选，如 ?lib=pig*）
  • 参数：Content-Type: application/json;charset=UTF-8

4. Job 和 Jobs

这些参数可以进行对 workflow, coordinator, 和 bundle jobs 进行提交，管理和信息的提取。

4.1 Standard Job Submission

  • 描述：job的提交
  • method：POST
  • url：/v1/jobs
  • 入参：xml作为payload，job的类取决于以下几个参数
    • oozie.wf.application.path : path to a workflow aplication directory, creates a workflow job
    • oozie.coord.application.path : path to a coordinator application file, creates a coordinator job
    • oozie.bundle.application.path : path to a bundle application file, creates a bundle job
  • 出参：Job的id

示例：

  • url：POST /v1/jobs
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
# xml的内容跟之前的job.properties一致

<?xml version="1.0" encoding="UTF-8"?>
<configuration>
	<property>
	   <name>user.name</name>
	   <value>icemimosa</value>
	</property>
	<property>
	   <name>nameNode</name>
	   <value>hdfs://localhost:9000</value>
	</property>
	<property>
	   <name>jobTracker</name>
	   <value>localhost:8032</value>
	</property>
	<property>
	   <name>queueName</name>
	   <value>default</value>
	</property>
	<property>
	   <name>examplesRoot</name>
	   <value>examples</value>
	</property>
	<property>
	   <name>outputDir</name>
	   <value>map-reduce</value>
	</property>
	<property>
	   <name>oozie.wf.application.path</name>
	   <value>hdfs://localhost:9000/user/icemimosa/examples/apps/map-reduce/workflow.xml</value>
	</property>
</configuration>
  • 返回：
1
2
3
4
5
6
# HTTP/1.1 201 CREATED
# Content-Type: application/json;charset=UTF-8

{
  "id": "0000001-160704142405284-oozie-icem-W"
}

注意：通过Oozie控制台, 上面的job状态处于 PREP 状态, 即 prepared。
Job PREP

  1. 启动方式一：提交Job的时候，url加上参数 ‘?action=start‘。 注意这个参数对 coordinator job 不起效。
  2. 启动方式二：提交完任务后，使用 /v1/job/[JOB-ID]?action=start (详细见4.6.2)

4.2 Proxy MapReduce Job Submission

通过这种方式可以直接提交只有一个MapReduce action节点的任务，从而避免去编写 workflow.xml 文件。
执行任务前需要确保对应的 Jars 和 文件 在 HDFS 上存在。

  • url：/v1/jobs?jobtype=mapreduce
  • method：POST

xml消息体中下面是需要指定的参数：

  • fs.default.name : The NameNode
  • mapred.job.tracker : The JobTracker (ResourceManager)
  • mapred.mapper.class : The map-task classname
  • mapred.reducer.class : The reducer-task classname
  • mapred.input.dir : The map-task input directory
  • mapred.output.dir : The reduce-task output directory
  • user.name : The username of the user submitting the job
  • oozie.libpath : A directory in HDFS that contains necessary Jars for your job
  • oozie.proxysubmission : Must be set to true

示例

  • url：POST /v1/jobs?jobtype=mapreduce
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
<?xml version="1.0" encoding="UTF-8"?>
<configuration>
    <property>
        <name>fs.default.name</name>
        <value>hdfs://localhost:9000</value>
    </property>
    <property>
        <name>mapred.job.tracker</name>
        <value>localhost:8032</value>
    </property>
    <property>
        <name>mapred.mapper.class</name>
        <value>org.apache.oozie.example.SampleMapper</value>
    </property>
    <property>
        <name>mapred.reducer.class</name>
        <value>org.apache.oozie.example.SampleReducer</value>
    </property>
    <property>
        <name>mapred.input.dir</name>
        <value>hdfs://localhost:9000/user/icemimosa/examples/input-data/text</value>
    </property>
    <property>
        <name>mapred.output.dir</name>
        <value>hdfs://localhost:9000/user/icemimosa/examples/output-data/map-reduce</value>
    </property>
    <property>
        <name>user.name</name>
        <value>icemimosa</value>
    </property>
    <property>
        <name>oozie.libpath</name>
        <value>hdfs://localhost:9000/user/icemimosa/examples/apps/map-reduce/lib</value>
    </property>
    <property>
        <name>oozie.proxysubmission</name>
        <value>true</value>
    </property>
</configuration>
  • 返回：
1
2
3
4
5
6
# HTTP/1.1 201 CREATED
# Content-Type: application/json;charset=UTF-8

{
  "id": "0000006-160704142405284-oozie-icem-W"
}

4.3 Proxy Pig Job Submission

同理，可以提交一个单节点的 Pig 节点。

  • url：/v1/jobs?jobtype=mapreduce
  • method：POST

xml 消息体中必须的参数

  • fs.default.name : The NameNode
  • mapred.job.tracker : The JobTracker
  • user.name : The username of the user submitting the job
  • oozie.pig.script : Contains the pig script you want to run (the actual script, not a file path)
  • oozie.libpath : A directory in HDFS that contains necessary Jars for your job
  • oozie.proxysubmission : Must be set to true

可选的参数

  • oozie.pig.script.params.size : The number of parameters you’ll be passing to Pig
  • oozie.pig.script.params.n : A parameter (variable definition for the script) in ‘key=value’ format, the ‘n’ should be an integer starting with 0 to indicate the parameter number
  • oozie.pig.options.size : The number of options you’ll be passing to Pig
  • oozie.pig.options.n : An argument to pass to Pig, the ‘n’ should be an integer starting with 0 to indicate the option number
1
oozie.pig.options.n ： 最后会被解释成执行脚本的 -D 参数。它还可以指定 -param_file 选项，用来指定 oozie.pig.script.params.n 中的参数。

示例：

1
2
3
4
# 指定 param_file 的文件

$ hadoop dfs -cat /user/icemimosa/pig_params.properties
 输出：INPUT=/user/icemimosa/examples/input-data/text
  • url：POST /v1/jobs?jobtype=pig
  • Content-Type: application/xml;charset=UTF-8
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
<?xml version="1.0" encoding="UTF-8"?>
<configuration>
    <property>
        <name>fs.default.name</name>
        <value>hdfs://localhost:9000</value>
    </property>
    <property>
        <name>mapred.job.tracker</name>
        <value>localhost:8032</value>
    </property>
    <property>
        <name>user.name</name>
        <value>icemimosa</value>
    </property>
    <property>
        <name>oozie.pig.script</name>
        <value>
            A  load '$INPUT' using PigStorage(':');
            B = foreach A generate $0 as id;
            store B into '$OUTPUT' USING PigStorage();
        </value>
    </property>
    <property>
        <name>oozie.pig.script.params.size</name>
        <value>1</value>
    </property>
    <property>
        <name>oozie.pig.script.params.0</name>
        <value>OUTPUT=/user/icemimosa/examples/output-data/pig</value>
    </property>
    <property>
        <name>oozie.pig.options.size</name>
        <value>2</value>
    </property>
    <property>
        <name>oozie.pig.options.0</name>
        <value>-param_file</value>
    </property>
    <property>
        <name>oozie.pig.options.1</name>
        <value>hdfs://localhost:9000/user/icemimosa/pig_params.properties</value>
    </property>
    <property>
        <name>oozie.libpath</name>
        <value>hdfs://localhost:9000/user/icemimosa/share/lib/pig</value>
    </property>
    <property>
        <name>oozie.proxysubmission</name>
        <value>true</value>
    </property>
</configuration>
  • 返回：
1
2
3
4
5
6
# HTTP/1.1 201 CREATED
# Content-Type: application/json;charset=UTF-8

{
	"id" : "job-3"
}

4.4 Proxy Hive Job Submission

与Pig任务类似。

  • url：/oozie/v1/jobs?jobtype=hive
  • method：POST

xml消息体中必须的参数

  • fs.default.name : The NameNode
  • mapred.job.tracker : The JobTracker
  • user.name : The username of the user submitting the job
  • oozie.hive.script : Contains the hive script you want to run (the actual script, not a file path)
  • oozie.libpath : A directory in HDFS that contains necessary Jars for your job
  • oozie.proxysubmission : Must be set to true

可选的参数

  • oozie.hive.script.params.size : The number of parameters you’ll be passing to Hive
  • oozie.hive.script.params.n : A parameter (variable definition for the script) in ‘key=value’ format, the ‘n’ should be an integer starting with 0 to indicate the parameter number
  • oozie.hive.options.size : The number of options you’ll be passing to Pig
  • oozie.hive.options.n : An argument to pass to Hive, the ‘n’ should be an integer starting with 0 to indicate the option number

示例：

  • url：POST /v1/jobs?jobtype=hive
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
<?xml version="1.0" encoding="UTF-8"?>
<configuration>
    <property>
        <name>fs.default.name</name>
        <value>hdfs://localhost:9000</value>
    </property>
    <property>
        <name>mapred.job.tracker</name>
        <value>localhost:8032</value>
    </property>
    <property>
        <name>user.name</name>
        <value>icemimosa</value>
    </property>
    <property>
        <name>oozie.hive.script</name>
        <value>
            CREATE EXTERNAL TABLE test (a INT) STORED AS TEXTFILE LOCATION '${INPUT}';
            INSERT OVERWRITE DIRECTORY '${OUTPUT}' SELECT * FROM test;
        </value>
    </property>
    <property>
        <name>oozie.hive.script.params.size</name>
        <value>2</value>
    </property>
    <property>
        <name>oozie.hive.script.params.0</name>
        <value>OUTPUT=/user/icemimosa/examples/output-data/hive</value>
    </property>
    <property>
        <name>oozie.hive.script.params.1</name>
        <value>INPUT=/user/icemimosa/examples/input-data/table</value>
    </property>
    <property>
        <name>oozie.libpath</name>
        <value>hdfs://localhost:9000/user/icemimosa/share/lib/hive</value>
    </property>
    <property>
        <name>oozie.proxysubmission</name>
        <value>true</value>
    </property>
</configuration>
  • 返回：
1
2
3
4
5
6
# HTTP/1.1 201 CREATED
# Content-Type: application/json;charset=UTF-8

{
	"id" : "job-3"
}

4.5 Proxy Sqoop Job Submission

执行一个Sqoop脚本任务。

  • url：/v1/jobs?jobtype=sqoop
  • method：POST

xml消息体中必须的参数

  • fs.default.name : The NameNode
  • mapred.job.tracker : The JobTracker
  • user.name : The username of the user submitting the job
  • oozie.sqoop.command : The sqoop command you want to run where each argument occupies one line or separated by “\n”
  • oozie.libpath : A directory in HDFS that contains necessary Jars for your job
  • oozie.proxysubmission : Must be set to true

可选的参数

  • oozie.sqoop.options.size : The number of options you’ll be passing to Sqoop Hadoop job
  • oozie.sqoop.options.n : An argument to pass to Sqoop hadoop job conf, the ‘n’ should be an integer starting with 0 to indicate the option number

示例：

  • url：POST /v1/jobs?jobtype=sqoop
  • Content-Type: application/xml;charset=UTF-8
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
<?xml version="1.0" encoding="UTF-8"?>
<configuration>
    <property>
        <name>fs.default.name</name>
        <value>hdfs://localhost:9000</value>
    </property>
    <property>
        <name>mapred.job.tracker</name>
        <value>localhost:8032</value>
    </property>
    <property>
        <name>user.name</name>
        <value>icemimosa</value>
    </property>
    <property>
        <name>oozie.sqoop.command</name>
        <value>
            import
            --connect
            jdbc:mysql://localhost:3306/oozie
            --username
            root
            --password
            anywhere
            --table
            WF_JOBS
            --target-dir
            /user/${wf:user()}/${examplesRoot}/output-data/sqoop
        </value>
    </property>
        <name>oozie.libpath</name>
        <value>hdfs://localhost:8020/user/icemimosa/share/lib/sqoop</value>
    </property>
    <property>
        <name>oozie.proxysubmission</name>
        <value>true</value>
    </property>
</configuration>
  • 返回：
1
2
3
4
5
6
# HTTP/1.1 201 CREATED
# Content-Type: application/json;charset=UTF-8

{
	"id" : "job-3"
}

4.6 Managing a Job

A HTTP PUT request ： starts, suspends, resumes, kills, update or dryruns a job

  • url：/v1/job/[JOB-ID]?action=Xxx

合法的 action 参数：start, suspend, resume, kill, dryrun, rerun, change 以及 ignore

4.6.1 Start a Job

  • 描述：启动一个job
  • method：PUT
  • url：/v1/job/[JOB-ID]?action=start
  • 返回：无返回（成功：HTTP/1.1 200 OK）

备注：这里仅能对Job状态为 PREP 的进行启动。

4.6.2 Re-Runing a Workflow Job

  • 描述：可以重新启动状态为 SUCCEEDED, KILLED or FAILED 的 workflow job
  • method：PUT
  • url：/v1/job/[JOB-ID]?action=rerun
  • 入参：
    • 与提交一个标准Job参数基本一致
    • 也可以指定跳过的action，PUT的xml消息体重配置中指定 oozie.wf.rerun.skip.nodes，值为action的name，多个以逗号 , 分隔
  • 返回：无返回（成功：HTTP/1.1 200 OK）

4.6.3 Re-Runing a coordinator job

  • 描述：可以重新启动状态为 RUNNING, SUCCEEDED, KILLED or FAILED的 coordinator job
  • method：PUT
  • url：/v1/job/[JOB-ID]?action=coord-rerun
  • 入参：
    • action=coord-rerun （必须）
    • type：值可以是 action 和 date
      • date：指定一个时间范围，时间范围用 :: 分隔
      • action：指定重启节点的范围，用 - 分隔
    • refresh：是否刷新input-events和output-events
    • nocleanp：是否清除output-events

示例：

  • url：
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
PUT /v1/job/job-3?action=coord-rerun&type=action&scope=1-2&refresh=false&nocleanup=false

## 或者
PUT /v1/job/job-3?action=coord-rerun&type=date&scope=2009-02-01T00:10Z::2009-03-01T00:10Z&refresh=false&nocleanup=false

## 返回
HTTP/1.1 200 OK
```	

### 4.6.4 Re-Runing a bundle job
- 描述：可以部分重新执行状态为 `RUNNING`, `SUCCEEDED` , `KILLED` or `FAILED`的 Coordinator Job。
- url：/v1/job/[JOB-ID]?action=bundle-rerun
- method：PUT
- 入参：
	- coord-scope：指定 Coordinator Job 的name （不是id）
	- date-scope：指定一个时间范围，时间范围用 `::` 分隔
	- refresh：是否刷新input-events和output-events
	- nocleanp：是否清除output-events
- 出参：无

> 示例

PUT /v1/job/job-3?action=bundle-rerun&coord-scope=coord-1&refresh=false&nocleanup=false

返回：

HTTP/1.1 200 OK

1
2
3
4
5
6
7
8
9

### 4.6.5 Changing endtime/concurrency/pausetime of a Coordinator Job
- 描述：可以修改一个状态 `不是KILLED` 的 Coordinator Job 的 `endtime`, `concurrency`, or `pausetime`
- url：/v1/job/job-3?action=change
- method：PUT
- 入参：`value` , 可以指定参数，且用分号 `;` 隔开
- 出参：无

> 示例

PUT /v1/job/job-3?action=change&value=endtime=2011-12-01T05:00Z;concurrency=100;pausetime=2011-12-01T05:00Z

返回：

HTTP/1.1 200 OK

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

### 4.6.6 Updating coordinator definition and properties
- 描述：可以替换现有 coordinator 定义文件
- url：/v2/job/[JOB-ID]?action=update
- method：PUT

### 4.6.7 Ignore a Coordinator Job or Action
- 描述：忽略 Coordinator Job 的一些Action的执行
- url：/v2/job/job-3?action=ignore
- method：PUT
- 入参：
	- type：目前只支持 action
	- scope：指定忽略的Action的ids范围
- 出参：无

> 示例：

PUT /v2/job/job-3?action=ignore&type=action&scope=3-4

1
2
3
4
5
6
7
8
9
10

## 4.7 Job Information
- 描述：获取Job的详情
- method：GET
- url：/v1/job/[JOB-ID]?show=info    (url中v1变为v2也支持)
- 入参：show=info（必须）
- 出参：Content-Type: application/json;charset=UTF-8
- 示例：
	- url：GET /v1/0000005-160704142405284-oozie-icem-W/?show=info
	- 返回：

{
  "appName": "map-reduce-wf",
  "externalId": null,
  "status": "SUCCEEDED",
  "group": null
  ......
  ......
}

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24

## 4.8 Job Application Definition
- 描述：获取 workflow, coordinator, bundle job的定义文件
- method：GET
- url：v1/job/[JOB-ID]?show=definition
- 入参：show=definition
- 出参：job的定义文件（xml格式）
- 示例：
	- url：GET v1/job/0000005-160704142405284-oozie-icem-W?show=definition
	- 返回：

	```xml
	## 截取部分（这里是workflow-app）
	<workflow-app xmlns="uri:oozie:workflow:0.2" name="map-reduce-wf">
	    <start to="mr-node"/>
	    <action name="mr-node">
	        <map-reduce>
	            <job-tracker>${jobTracker}</job-tracker>
	            <name-node>${nameNode}</name-node>
	            <prepare>
	                <delete path="${nameNode}/user/${wf:user()}/${examplesRoot}/output-data/${outputDir}"/>
	            </prepare>
	            ......
	            ......

4.9 Job Log

  • 描述：获取job的日志
  • method：GET
  • url：/v1/job/[JOB-ID]?show=log
  • 入参：
    • show=log (必须)
    • logfilter (可选，过滤日志的参数，格式是 opt1=val1;opt2=val1;opt3=val1)
      • recent：指定最近小时/分钟的日志（如3h, 30m）
      • start：开始时间（yyyy-MM-dd HH:mm:ss,SSS）
      • end：结束时间
      • loglevel：可用竖线 | 分隔，可选 ALL, DEBUG, ERROR, INFO, TRACE, WARN, FATAL
      • text：搜索log的参数
      • limit：日志的数量
      • debug：打印debug日志
      • 参考：日志 logfilter 参数
  • 出参：text/plain格式

  • 示例：

    • url：/v1/job/0000005-160704142405284-oozie-icem-W?show=log&logfilter=limit=8;loglevel=INFO

    • 返回：

      1
      2
      3
      4
      5
      6
      2016-07-04 19:15:17,992  INFO ActionStartXCommand:520 - SERVER[IceMimosa.local] USER[icemimosa] GROUP[-] TOKEN[] APP[map-reduce-wf] JOB[0000005-160704142405284-oozie-icem-W] ACTION[0000005-160704142405284-oozie-icem-W@:start:] Start action [0000005-160704142405284-oozie-icem-W@:start:] with user-retry state : userRetryCount [0], userRetryMax [0], userRetryInterval [10]
      2016-07-04 19:15:17,995  INFO ActionStartXCommand:520 - SERVER[IceMimosa.local] USER[icemimosa] GROUP[-] TOKEN[] APP[map-reduce-wf] JOB[0000005-160704142405284-oozie-icem-W] ACTION[0000005-160704142405284-oozie-icem-W@:start:] [***0000005-160704142405284-oozie-icem-W@:start:***]Action status=DONE
      2016-07-04 19:15:17,996  INFO ActionStartXCommand:520 - SERVER[IceMimosa.local] USER[icemimosa] GROUP[-] TOKEN[] APP[map-reduce-wf] JOB[0000005-160704142405284-oozie-icem-W] ACTION[0000005-160704142405284-oozie-icem-W@:start:] [***0000005-160704142405284-oozie-icem-W@:start:***]Action updated in DB!
      2016-07-04 19:15:18,090  INFO ActionStartXCommand:520 - SERVER[IceMimosa.local] USER[icemimosa] GROUP[-] TOKEN[] APP[map-reduce-wf] JOB[0000005-160704142405284-oozie-icem-W] ACTION[0000005-160704142405284-oozie-icem-W@mr-node] Start action 
      ......
      ......

4.10 Job graph

  • 描述：获取job的视图
  • method：GET
  • url：/v1/job/[JOB-ID]?show=graph
  • 入参：
    • show：值为graph （必须）
    • show-kill：是否显示kill节点, true or false
  • 出参：类型 image/png

  • 示例：

    • url：GET /v1/job/0000005-160704142405284-oozie-icem-W?show=graph&show-kill=true

    • 返回：

      Job Graph

4.11 Jobs Information

  • 描述：获取Jobs的信息
  • method：GET
  • url：/v1/jobs
  • 入参：
    • filter：值为user=Xxx，进行用户过滤, 多个用 ; 分隔
      • name：application name from the workflow/coordinator/bundle definition
      • user：the user that submitted the job
      • group：the group for the job
      • status： the status of the job
      • jobtype：值为 wf , coordinator or bundle
      • 备注：不同的属性名称用 AND 过滤，相同的属性名称用 OR 过滤。
    • offset：分页第一个下标（从1开始），默认1
    • len：获取的数量，默认50

  • 示例：

    • url：GET /v1/jobs?filter=user=icemimosa&offset=1&len=2

    • 返回：

      1
      2
      3
      4
      5
      6
      7
      8
      9
      10
      11
      12
      13
      14
      15
      16
      17
      18
      19
      20
      21
      22
      23
      24
      25
      26
      27
      28
      29
      30
      31
      32
      33
      34
      35
      36
      37
      38
      39
      40
      41
      42
      43
      44
      45
      46
      47
      	{
      	  "total": 14,
      	  "offset": 1,
      	  "len": 2,
      	  "workflows": [
      	    {
      	      "appName": "oozie-mapreduce",
      	      "externalId": null,
      	      "conf": null,
      	      "run": 0,
      	      "acl": null,
      	      "appPath": null,
      	      "parentId": null,
      	      "lastModTime": "Tue, 05 Jul 2016 01:59:30 GMT",
      	      "consoleUrl": "http://IceMimosa.local:11000/oozie?job=0000007-160704142405284-oozie-icem-W",
      	      "createdTime": "Tue, 05 Jul 2016 01:59:19 GMT",
      	      "startTime": "Tue, 05 Jul 2016 01:59:19 GMT",
      	      "toString": "Workflow id[0000007-160704142405284-oozie-icem-W] status[KILLED]",
      	      "id": "0000007-160704142405284-oozie-icem-W",
      	      "endTime": "Tue, 05 Jul 2016 01:59:30 GMT",
      	      "user": "icemimosa",
      	      "actions": [],
      	      "status": "KILLED",
      	      "group": null
      	    },
      	    {
      	      "appName": "oozie-mapreduce",
      	      "externalId": null,
      	      "conf": null,
      	      "run": 0,
      	      "acl": null,
      	      "appPath": null,
      	      "parentId": null,
      	      "lastModTime": "Tue, 05 Jul 2016 01:57:30 GMT",
      	      "consoleUrl": "http://IceMimosa.local:11000/oozie?job=0000006-160704142405284-oozie-icem-W",
      	      "createdTime": "Tue, 05 Jul 2016 01:57:29 GMT",
      	      "startTime": "Tue, 05 Jul 2016 01:57:29 GMT",
      	      "toString": "Workflow id[0000006-160704142405284-oozie-icem-W] status[SUSPENDED]",
      	      "id": "0000006-160704142405284-oozie-icem-W",
      	      "endTime": null,
      	      "user": "icemimosa",
      	      "actions": [],
      	      "status": "SUSPENDED",
      	      "group": null
      	    }
      	  ]
      }

4.12 Jobs information using Bulk API

获取一个指定 bundle 的一个大的Actions集合。这有利于监控所有Coordinator Jobs的状态。查询的时候可以指定 bulk ，offset 和 len 参数。

在进行查询的时候可以指定如下的bulk参数：

  • bundle : the application name from the bundle definition
  • coordinators : the application name(s) from the coordinator definition.
  • actionStatus : the status of coordinator action (Valid values are WAITING, READY, SUBMITTED, RUNNING, SUSPENDED, TIMEDOUT, SUCCEEDED, KILLED, FAILED)
  • startCreatedTime : the start of the window you want to look at, of the actions’ created time
  • endCreatedTime : the end of above window
  • startScheduledTime : the start of the window you want to look at, of the actions’ scheduled i.e. nominal time.
  • endScheduledTime : the end of above window

参数 Tips：

  1. bundle必须指定，所有的参数都是进行 AND 查询。
  2. actionStatus如果未指定，默认为 KILLED,FAILED。
  3. bulk的这些参数使用分号 ; 进行分割。
  4. coordinators 和 actionStatus 如果指定多个可以使用逗号 , 分隔进行 OR 查询。
  5. 时间使用 UTC 的格式。

示例：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
GET /v1/jobs?bulk=bundle=my-bundle-app;coordinators=my-coord-1,my-coord-5;actionStatus=KILLED&offset=1&len=50

## 返回
{
  offset: 1,
  len: 50,
  total: 1002,
  	bulkresponses: [
  	{
      bulkbundle:
      {
        bundleJobName: "my-bundle-app",
        bundleJobId: "0-200905191240-oozie-B",
        status: "SUSPENDED",
      },
      bulkcoord:
      {
        coordJobName: "my-coord-1",
        status: "SUSPENDED",
      },
      bulkaction:
      {
        id: "0-200905191240-oozie-C@21",
        coordJobId: "0-200905191240-oozie-C",
        actionNumber: 21,
        externalId: "job_00076_0009",
        status: "KILLED",
        externalStatus: "FAILED",
        errorCode: "E0902",
        errorMessage: "Input file corrupt",
        createdTime: "Fri, 02 Jan 2009 00:00:00 GMT",
        nominalTime: "Thu, 01 Jan 2009 00:00:00 GMT",
        missingDependencies: "hdfs://nn:port/user/joe/file.txt"
      },
    },
    {
      bulkbundle:
      {
        bundleJobName: "my-bundle-app",
        bundleJobId: "0-200905191240-oozie-B",
        status: "SUSPENDED",
      }
      bulkcoord:
      {
        coordJobName: "my-coord-5",
        status: "SUSPENDED",
      }
      bulkaction:
      {
        id: "0-200905191245-oozie-C@114",
        coordJobId: "0-200905191245-oozie-C",
        actionNumber: 114,
        externalId: "job_00076_0265",
        status: "KILLED",
        externalStatus: "KILLED",
        errorCode: "E0603",
        errorMessage: "SQL error in operation ...",
        createdTime: "Fri, 02 Jan 2009 00:00:00 GMT",
        nominalTime: "Thu, 01 Jan 2009 00:00:00 GMT",
        missingDependencies:
      }
    }
    ...
  ]
}
文章目录
  1. Oozie-6_Web Service API
  2. 1. 简介
  3. 2. versions
  4. 3. admin
    1. 3.1 status
    2. 3.2 available-oozie-servers
    3. 3.3 OS Environment
    4. 3.4 Java System Properties
    5. 3.5 Oozie Configuration
    6. 3.6 Oozie Instrumentation 和 Oozie Metrics
    7. 3.7 Build Version
    8. 3.8 Available Time Zones
    9. 3.9 List available sharelib
  5. 4. Job 和 Jobs
    1. 4.1 Standard Job Submission
    2. 4.2 Proxy MapReduce Job Submission
    3. 4.3 Proxy Pig Job Submission
    4. 4.4 Proxy Hive Job Submission
    5. 4.5 Proxy Sqoop Job Submission
    6. 4.6 Managing a Job
      1. 4.6.1 Start a Job
      2. 4.6.2 Re-Runing a Workflow Job
      3. 4.6.3 Re-Runing a coordinator job
    7. 返回：
    8. 返回：
    9. 4.9 Job Log
    10. 4.10 Job graph
    11. 4.11 Jobs Information
    12. 4.12 Jobs information using Bulk API